Skip to content

fix(process-engine): survive optimistic-lock races on task, process, and context rows#151

Merged
alsergs merged 4 commits into
mainfrom
fix/process-engine-version-races
Jul 24, 2026
Merged

fix(process-engine): survive optimistic-lock races on task, process, and context rows#151
alsergs merged 4 commits into
mainfrom
fix/process-engine-version-races

Conversation

@kichasov

Copy link
Copy Markdown
Collaborator

Why

The dbaas-aggregator's backup flow keeps failing under load with VersionMismatchException from BackupDatabaseTask (four DBaaS IT runs bitten so far; diagnostics in Netcracker/qubership-dbaas runs 29988579829 and 30026600214):

Scheduler: Unhandled exception VersionMismatchException: 'Version in repository: 2, Task Version 1'
           during execution of task ...BackupDatabaseTask. Treating as failure.
ExecutePicked: Failed while completing execution ... Execution will likely remain scheduled and locked/picked.

Several actors hold independent in-memory copies of the same pe_task_instance / po_context / process rows — the POProcess tick (every 2 s), the executing task, ProcessTaskFailureHandler, and the TaskExecutorService timeout watchdog. Whoever loses a milliseconds-wide race aborts its completion; the db-scheduler execution then stays picked until dead-execution revival, and externally that is a hung backup and an HTTP 500.

Two defects compound it: the repositories used a check-then-act version compare (SELECT, compare in Java, unguarded UPDATE), which under true concurrency also silently loses one of two writes; and TaskExecutionWrapper ran its completion callback twice on failure.

What

  • Atomic optimistic locking: all three repositories guard the UPDATE itself (WHERE ... AND version=?, affected-row check) and bump the in-memory version only after success. Exception contract unchanged.
  • Must-win writes recover instead of aborting: terminal COMPLETED/FAILED saves go through new TaskInstanceImpl.saveResolvingConflict(reapply) — on a conflict the row is reloaded, the idempotent mutation re-applied, and the fresh copy saved (used in AbstractProcessTask, ProcessTaskFailureHandler, AsyncTaskWithPolling, the timeout watchdog). DataContext.apply does the same for context mutations, which covers task start/end times and state descriptions.
  • TaskExecutionWrapper runs its callback exactly once and logs the swallowed error.

A completion that today crashes with a stuck execution would, with slightly different timing, have succeeded anyway — the recovery path persists exactly that outcome, so no new interleavings are introduced.

How to verify

mvn -pl core-process-orchestrator test — 37 tests green, including the new VersionConflictTest: stale-copy failures for all three repositories (winning write survives, no corruption), terminal-save recovery via reload, context apply recovery preserving the concurrent writer's data, and single-callback wrapper behavior.

After merge, the consumer side (Netcracker/qubership-dbaas) needs a process-engine bump (currently pinned to 1.5.2) plus switching AbstractDbaaSTask.updateState to context.apply(...) so its status-description writes get the same recovery — I will follow up there and re-run the DBaaS IT that reproduces the race.

…and context rows

Under load the aggregator's backup flow kept dying with
VersionMismatchException ('Version in repository: 2, Task Version 1' /
'Current version are less than saved') from BackupDatabaseTask: several
actors hold independent in-memory copies of the same row — the POProcess
tick (every 2s), the executing task, ProcessTaskFailureHandler, and the
TaskExecutorService timeout watchdog — and the loser of a
milliseconds-wide race aborted its completion, leaving the db-scheduler
execution stuck picked until dead-execution revival. Externally that
surfaced as hung backups and HTTP 500s.

Three fixes:
- All three repositories now guard the UPDATE itself (WHERE version=?)
  instead of a check-then-act compare, closing the silent lost-update
  window, and bump the in-memory version only on success.
- Must-win writes recover instead of aborting: terminal COMPLETED/FAILED
  saves go through TaskInstanceImpl.saveResolvingConflict (reload,
  re-apply, save), and DataContext.apply re-applies its idempotent
  mutation on a fresh copy after a conflict.
- TaskExecutionWrapper ran its callback twice on failure (catch plus
  finally); it now runs exactly once and logs the error.
@kichasov
kichasov requested a review from lis0x90 as a code owner July 23, 2026 17:54
@github-actions github-actions Bot added the bug Something isn't working label Jul 23, 2026
kichasov added 2 commits July 23, 2026 21:06
…after recovery

Review findings on the race fix:

- Process-level terminal writes (POProcess completion FAILED/COMPLETED,
  AsyncTaskWithPolling failure, the timeout watchdog) still called raw
  save() and could abort on the same version race the task writes were
  protected from. ProcessInstanceImpl gains the same
  saveResolvingConflict, used at all four sites; the pre-existing
  reload() is reused.
- After conflict recovery both saveResolvingConflict variants and
  DataContext.apply now sync the caller's instance with the persisted
  state (contents, fields, version, clean dirty flag). Without that the
  caller kept a stale copy and the very next save() — for example the
  trailing process save in the POProcess completion handler — failed
  again despite the recovery having succeeded.
- FutureKey.equals no longer accepts String/UUID (equals-contract violation);
  terminate() now filters by getTaskId() explicitly (S2159)
- re-interrupt the thread on InterruptedException in the watchdog and in
  terminate()'s wait loop (S2142)
- terminate() returns Future<Void> instead of Future<?> (S1452)
- TaskExecutionWrapper catches Exception, not Throwable (S1181)
- ProcessTaskFailureHandler reuses a single Optional instance (S3655)
- suppress S3011 (reflective access into db-scheduler internals) and
  S2160 (HashMap content equality is intentional) with justifications
@kichasov
kichasov force-pushed the fix/process-engine-version-races branch from 7a8da29 to 36b4a12 Compare July 23, 2026 19:02
An interrupt is the watchdog's normal shutdown signal: the completion
callback in execute() cancels it with cancel(true) once the task finishes
in time. The shared catch treated that interrupt as a timeout and marked
the completed task and its process FAILED — and saveResolvingConflict made
the bogus overwrite reliable instead of losing the version race.

Split the catch: InterruptedException restores the interrupt flag and
stops watching without touching state; only ExecutionException and
TimeoutException cancel the worker and mark the task and process FAILED.
The watchdog body moved to a package-private watchTask() so tests can
drive both paths directly.
@kichasov
kichasov requested a review from Logaka July 23, 2026 19:35
@sonarqubecloud

Copy link
Copy Markdown

@alsergs
alsergs merged commit 802dc4a into main Jul 24, 2026
8 checks passed
@alsergs
alsergs deleted the fix/process-engine-version-races branch July 24, 2026 09:13
kichasov added a commit to Netcracker/qubership-dbaas that referenced this pull request Jul 24, 2026
…ck race fixes (#603)

Picks up Netcracker/qubership-core-java-libs#151, merged to main and
published as 1.5.5-20260724.091358-10: atomic version guards on the task,
process, and context rows, conflict-recovering terminal saves, a single
completion callback per task, and a watchdog that no longer fails a task
that completed in time.

AbstractDbaaSTask.updateState switches from put+save to context.apply so
its status-description writes get the same conflict recovery; a plain
save() aborted the whole task when the POProcess tick or the timeout
watchdog wrote the same row first.

Verified on verify/process-engine-race-fix and verify/pe-race-fix-nb-status
against a throwaway snapshot of the same code: the DBaaS IT concurrent
section passes and the VersionMismatchException signature is gone from the
aggregator logs.
Logaka pushed a commit to Netcracker/qubership-dbaas that referenced this pull request Jul 24, 2026
* chore(test-apps): bump dbaas client pins to refreshed mounted-secret snapshots

Point the sample services at the mounted-secret libraries rebuilt on top of master:
- spring: dbaas-client 9.1.3 -> 9.1.5-mounted-secret-SNAPSHOT
- quarkus: dbaas-quarkus 10.1.1 -> 10.1.3-mounted-secret-SNAPSHOT
- go: dbaas-base-client -> v3.5.4-0.20260701125619-f0a56365482c (feat/mounted-secret-provider
  merged with main), core-lib-go 3.11.0 -> 3.12.0, go directive 1.26.4 (required by deps).

* docs(operator): normalize operator docs and comments to US English (#539)

Apply the english-us-developer-style skill: normalize British spellings to American across the
operator's user-facing docs, the Grafana dashboard description, metric Help strings, AGENTS.md,
and code/dev/test comments; drop one filler word. Text-only, no logic changes.

* test(sample-apps): added sample apps to test dbaas clients

* test(sample-apps): add Phase C — direct M2M REST to the aggregator

Add a third integration pass where all three sample clients (go/spring/quarkus) call
the DBaaS aggregator directly over REST, authenticated with a Kubernetes projected
service-account token (audience `dbaas`), instead of via the dbaas-agent sidecar.

- helm (3 charts): dedicated ServiceAccount per app; a `dbaas`-audience projected token
  volume at /var/run/secrets/tokens/dbaas + KUBERNETES_M2M_ENABLED, gated on a new
  M2M_ENABLED value (default false — Pass A/B unchanged). M2M is activated without
  touching app sources, via helm env overrides: Spring flips
  DBAAS_RESTCLIENT_RESTTEMPLATE_BASIC_AUTH=false (selects the M2M OkHttp client);
  Quarkus clears the aggregator basic-auth creds (DbaasClientProducer picks the M2M
  client); Go gains API_DBAAS_ADDRESS (koanf api.dbaas.address).
- aggregator: map the three sample-app ServiceAccounts to DB_CLIENT in
  service-account-roles.yaml.
- deploy-sample-services.sh: optional third arg M2M_ENABLED, threaded to all charts.
- workflow: grant the aggregator SA system:service-account-issuer-discovery (for OIDC
  token verification) + restart; add the Phase C deploy/wait/test block after Pass B
  (the phase-agnostic IT suite is reused unchanged).
- docs: document Pass A/B/C + the M2M env in the three sample READMEs.

* fix(spring-test-app): add dummy M2MManager for direct-M2M phase

In Phase C the dbaas Spring client uses its M2M OkHttp flavor (basic-auth=false), whose
dbaasRestClient bean autowires an M2MManager the app didn't provide, so it failed to start.
Add a DummyM2MManager @bean (guarded by @ConditionalOnMissingBean). For direct dbaas calls the
aggregator is authenticated with the pod's projected dbaas-audience token, not this manager's
token, so the dummy is sufficient; the bean is unused in the basic-auth (Pass A/B) phases.

* test(spring-test-app): add Tenant header to M2M dbaas client

The dbaas Spring M2M flavor is an OkHttp client with no context-propagation interceptor (unlike
the RestTemplate/Quarkus transports), so the Tenant header is dropped and tenant-scoped
GetOrCreateDatabase fails the aggregator's classifier-vs-request tenant check (CORE-DBAAS-4054),
which only runs for K8s-JWT (M2M) callers. Decorate the dbaasRestClient bean (via a BeanPostProcessor)
to re-add the Tenant header from the current TenantContext. Temporary until OkHttp context
propagation lands in the dbaas client library.

* feat: CR conversion skill for DBaaS  (#544)

* feat: CR conversion skill for DBaaS

* fix: removing the wrong file

* fix: non chart sources now pass explicit values

* chore: add APM agent support (#556)

* chore(test-apps): drop redundant sample-app resource profiles (#554)

Each sample-app chart shipped resource-profiles/{dev,dev-ha,prod,prod-nonha}.yaml,
but all four files are identical and their five keys (CPU/MEMORY request/limit,
REPLICAS) already match the values.yaml defaults. The deploy script overlaid
resource-profiles/dev.yaml with `-f`, which changed nothing — `helm template`
renders byte-for-byte identically with and without it.

Remove the resource-profiles directories from all three charts and drop the
`-f .../resource-profiles/dev.yaml` overlay from deploy-sample-services.sh.
values.yaml already holds the resource and replica settings, so the rendered
manifests are unchanged. These are throwaway CI test apps; the multi-tier
profile convention adds no value here.

* chore(ci): skip operator build for docs/dev-only commits (#553)

The Go operator build ran on every push to main and every PR, including
commits that touch only dbaas-operator/docs or dbaas-operator/dev — neither
of which affects the built binary.

Add paths-ignore for dbaas-operator/docs/** and dbaas-operator/dev/** to both
the push and pull_request triggers. paths-ignore skips the run only when every
changed file matches, so a mixed docs-plus-code commit still builds. Neither
main nor feat/operator-dev requires Go Build as a status check, so a docs-only
PR is not blocked by a skipped run.

* chore(ci): align log artifact names, add sample-service diagnostics (#552)

Both log artifacts read ambiguously (each looked like "just logs"). Rename
them to say what each holds and which one to download:

  dbaas-pod-logs           -> dbaas-pod-stdout         (Fluent Bit stdout)
  dbaas-logs               -> dbaas-k8s-diagnostics    (kubectl describe/events/previous)
  sample-services-pod-logs -> sample-services-pod-stdout

The sample-service workflow only exposed Fluent Bit stdout as an artifact;
its kubectl diagnostics went to the job log only. Promote that dump to a
downloadable sample-services-k8s-diagnostics artifact (same snapshot still
folded into the console via ::group::), collected only on failure. That is
the signal that matters when a pod never starts (ImagePull/CrashLoop/
OOMKilled), where the Fluent Bit stdout artifact is empty.

Step names aligned to "Collect/Upload ... (tool)" across both workflows.

* feat: skill for transforming DBaaS runtime provisioning (#546)

* feat: CR conversion skill for DBaaS

* fix: removing the wrong file

* fix: non chart sources now pass explicit values

* feat: first draft

* fix: pushing fixes and review changes

* fix: review changes

* fix: review change

* fix(ci): configure default S3 alias for the backup daemon (#593)

Backup integration tests send `storageName: "default"` in every backup and
restore request. The postgres backup daemon only resolves a storage name when
S3 aliases are enabled, and rejects the request otherwise:

    storageName was provided but S3 aliases are not enabled

The adapter turns that response into a panic and returns HTTP 500, so the
aggregator marks the logical backup as RETRYABLE_FAIL and every
BackupV3IT.Postgresql backup test fails.

The pgskipper-operator chart enables aliases only when `backupDaemon.s3Aliases`
is a non-empty list: an empty list leaves `s3AliasesUsed` unset in the CR, so
the daemon starts without `S3_ALIASES_USED=true`. Declare one alias named
`default` that points at the same MinIO tenant already configured under
`s3Storage`.

* fix(tests): read adapter credentials from the mounted secret (#594)

* fix(ci): configure default S3 alias for the backup daemon

Backup integration tests send `storageName: "default"` in every backup and
restore request. The postgres backup daemon only resolves a storage name when
S3 aliases are enabled, and rejects the request otherwise:

    storageName was provided but S3 aliases are not enabled

The adapter turns that response into a panic and returns HTTP 500, so the
aggregator marks the logical backup as RETRYABLE_FAIL and every
BackupV3IT.Postgresql backup test fails.

The pgskipper-operator chart enables aliases only when `backupDaemon.s3Aliases`
is a non-empty list: an empty list leaves `s3AliasesUsed` unset in the CR, so
the daemon starts without `S3_ALIASES_USED=true`. Declare one alias named
`default` that points at the same MinIO tenant already configured under
`s3Storage`.

* fix(tests): read adapter credentials from the mounted secret

The postgres adapter no longer publishes its basic-auth credentials through
DBAAS_ADAPTER_API_USER and DBAAS_ADAPTER_API_PASSWORD. Since pgskipper-operator
moved the adapter to mounted secrets, the credentials live in the
`dbaas-adapter-credentials` secret, mounted at
/var/run/secrets/postgresql/dbaas-adapter-credentials.

MigrationHelper still fell back to `dbaas-aggregator-credentials.v1`, which is
the mongo adapter's secret and does not exist in the postgres namespace. Every
test that creates a database through the adapter API failed with:

    Key 'username' not found in secret 'dbaas-aggregator-credentials.v1'
    in namespace postgres

Resolve the secret from the pod spec: find the volume mount whose path ends with
`dbaas-adapter-credentials` and read the backing secret. Taking the name from the
pod rather than hard-coding it keeps the helper working across adapters. The
environment-variable lookup stays first, and the aggregator secret stays as the
last fallback, so adapters on older versions keep working.

* fix(aggregator): stop holding the repository mutex across Postgres writes (#597)

DBAAS_REPOSITORIES_MUTEX is a single process-wide monitor that guards the
in-memory H2 cache. Three write paths held it across a Postgres operation:
DatabaseRegistryDbaasRepository.save (a Hibernate flush), and the two deleteById
/ delete methods (a Postgres delete). That created a deadlock spanning two
layers.

One thread would update a row, take its row lock, then block entering the
monitor; the monitor holder, inside its own transaction, would flush an UPDATE
that needed the same row and wait on the row lock. Neither side can detect it:
Postgres sees the blocked thread's transaction as idle, and the JVM sees the
monitor holder as RUNNABLE in a native socket read. The aggregator froze every
mutex-guarded path until the pod restarted, surfacing as flaky OperatorIT and
BlueGreenV1IT timeouts. Captured with thread dumps and pg_stat_activity in the
integration-tests workflow.

Do the Postgres write outside the monitor and hold the monitor only for the H2
mirror, matching PhysicalDatabaseDbaasRepository.delete and
DatabaseDbaasRepository.deleteAll, which already follow this pattern. save now
takes no monitor at all, like PhysicalDatabaseDbaasRepository.save, because its
block is Postgres-only; the H2 mirror is refreshed asynchronously by the
classifier table listener.

* build(operator): generate the chart's CRD templates and verify them in CI (#590)

* build(operator): generate the chart's CRD templates and verify them in CI

The chart carries its own copy of every CRD so it can gate them on
DBAAS_OPERATOR_ENABLED, and those copies were maintained by hand. They drifted.

The chart shipped shortName "dbdp" for DatabaseAccessPolicy while the marker in
api/v1 — unchanged since the resource was introduced — says "dbdap". Every
documented `kubectl get dbdap` therefore failed against a chart-installed
cluster. Three more templates carried stale descriptions, so `kubectl explain`
answered from an older version of the API.

Each template turned out to be exactly the generated CRD wrapped in the
conditional and nothing else: all eight have their only two Helm expressions on
the first and last line. That makes the copy mechanical, so add
`make sync-helm-crds` to produce it, and depend on `manifests` so it can never
be taken from a stale config/crd/bases.

Add a verify-generated job that reruns manifests, generate and sync-helm-crds
and fails on any diff. It catches the two mistakes that caused this — forgetting
to regenerate, and forgetting to copy — and it would have caught the shortName
divergence the day it appeared.

Mark both CRD trees linguist-generated and record the target in AGENTS.md, so
the next person edits the markers rather than the output.

* fix(operator): delete orphan CRD templates and catch untracked drift

sync-helm-crds only overwrote templates for CRDs that still exist, so
removing a CRD from the API left its chart template behind and the chart
kept installing it. Remove the generated crd-*.yaml files before the
copy loop so the sync carries deletions too.

The verify-generated job used git diff, which ignores untracked files:
a brand-new CRD whose generated artifacts were never committed passed
the check silently. Gate on git status --porcelain instead.

* refactor(operator): make conditions the status contract, phase a summary (#585)

* refactor(operator): make conditions the status contract, phase a summary

The Kubernetes API conventions deprecate `phase` as a state model: "The pattern
of using phase is deprecated. Newer API types should use conditions instead."
The operator already models state correctly with Ready/Stalled conditions, but
phase was still an input to control flow, and a closed OpenAPI enum on the
status field made it an upgrade hazard.

Stop reading phase in controller logic. Six predicates asked "is this terminal?"
by comparing phase against Succeeded/InvalidConfiguration; they now derive the
same answer from the conditions via isTerminal(). The DatabaseSecretClaim
safety-net trigger mixed both mechanisms (root observedGeneration plus phase)
and now uses isReadyForGeneration(), which reads the Ready condition's own
observedGeneration — the field that exists for exactly this purpose.

Drop the enum and the default from status.phase. A closed enum on a status
field means that shipping a new phase value before the updated CRD reaches the
cluster makes the API server reject the entire status write, conditions
included, leaving the resource unobservable. Defaulting is equally wrong here:
status is owned by the controller, which always sets phase alongside the
conditions.

Phase itself stays. It is the only way to show one readable column in
`kubectl get`, because JSONPath can select but not branch. It is now documented
as an observational summary, and a Ready column is added next to it so the
authoritative signal is visible too.

No behavior change: the predicates are equivalent, which the existing controller
suite confirms, and new unit tests pin the equivalence.

* fix(operator): make the terminal predicate generation-aware

Conditions persist across reconciles, so after a spec change a reconcile
that exits early (for example on a benign Secret create/update race)
still carries the previous generation's Ready=True. isTerminal counted
it as terminal, letting patchStatusOnExit stamp status.observedGeneration
for a spec the controller never finished processing. The former
phase-based predicate was immune: phase was reset to Processing at the
start of every reconcile.

Require the terminal condition's observedGeneration >= obj.Generation,
pass the generation at all five shouldObserve sites, and add stale-Ready
and stale-Stalled regression tests.

* chore(operator): enforce balancing-rule name and blank values in the CRD (#587)

The three balancing-rule CRDs carried no declarative validation at all, so
every constraint was checked by the controller after the object already
existed: a mistyped CR was admitted, reconciled, moved to InvalidConfiguration
and left behind for someone to notice and delete. Two of those constraints are
trivially expressible in the schema, and one of them is not fully covered by
the controller either.

Enforce the singleton name with CEL, matching what NamespaceBinding already
does. A balancing rule is a singleton per namespace, and the name is fixed;
rejecting a wrong name at admission gives the user the error in kubectl apply
instead of a doomed object.

Reject whitespace-only strings with Pattern=\S on type, dbType, name and
physicalDatabaseId, and on the microservices and namespaces items. MinLength=1
does not cover this: " " has length 1 and passes. The controller's TrimSpace
checks caught it, but only after admission.

The controller-side checks stay as they are. They cover what the schema cannot
express — uniqueness over the flattened (type x microservice) product, the
operator-namespace check that depends on runtime config — and they keep working
against a stale CRD.

One existing spec created a second NamespaceBalancingRule under a non-singleton
name to set up a rule-name collision. That shortcut no longer passes admission,
so the spec now creates the conflicting singleton in another namespace, which is
also the real scenario: aggregator rule names are global and each namespace has
exactly one singleton.

* chore(operator): align identifier casing with Go and API conventions (#588)

* chore(operator): align identifier casing with Go and API conventions

The CR types deliberately mirror the aggregator's Java DTOs, so the JSON names
follow Java camelCase: tenantId, physicalDatabaseId, lastRequestId. That is a
documented convention and it stays. One field broke it.

Rename the status field trackingID to trackingId. The operator described the
same value two ways: the aggregator client already used `json:"trackingId"`,
while the CR status used trackingID. The wire and the CR now agree, and the
field matches the four other Id-suffixed names in the API. Nothing reads the
field by its JSON name in tests, so the change is confined to the tag, the
regenerated CRD, and the docs — including a kubectl jsonpath example that would
otherwise have silently returned nothing.

Fix the Go identifiers that violate the initialism rule: TenantId, DbName and
DbType become TenantID, DBName and DBType, and three stray Id struct fields
become ID. JSON tags are untouched, so this changes no API surface.

Enable revive's var-naming rule to keep it that way. It immediately found the
three Id fields that manual review had missed. The rule covers Go identifiers
only; the camelCase JSON tags are unaffected.

One case is deliberately left alone: tmf.Response.Id in the aggregator mock is
a field of an external library type.

* chore(operator): rename DbType in tests merged from feat/operator-dev

The balancing-rule validation tests landed in the base branch with the
pre-rename field name and missed the DBType sweep.

* docs(operator): record why the required label cannot be validated by CEL (#589)

The comment on DatabaseSecretClaim said CEL validation of metadata.labels "is
not supported by controller-gen". That is not where the limit lives, and the
wording sends the next reader down a dead end: controller-gen generates such a
rule without complaint, and the CRD only fails when the API server compiles it.

A root-level XValidation rule sees a trimmed metadata — name and generateName
only. Referencing self.metadata.labels makes the API server reject the entire
CRD with "undefined field 'labels'"; self.metadata.namespace fails the same way,
which also rules out a CEL check that spec.classifier.namespace matches
metadata.namespace. Both were verified against a live API server.

Record the real reason and both consequences, so the controller-side pre-flight
checks read as a deliberate choice rather than an oversight.

The regenerated CRD also picks up description drift that had accumulated between
config/crd/bases and the hand-maintained Helm template for this resource.

* fix: adding the finalizer permission for secret claim (#586)

* fix: adding the finalizer permission for secret claim

* fix: aligning with the cluster role changes

* feat: enforcing owner ref RBAC in kind envs

* docs(operator): add databasesecretclaims/finalizers to the permission reference

The RBAC rule landed without its row in the permission table, which
claims to explain why each permission is needed.

---------

Co-authored-by: Roman Kichasov <roman.kichasov.qubership@gmail.com>

* fix(ci): pin kind v0.32.0 for kubeadm v1beta4 config

helm/kind-action@v1.14.0 defaults to kind v0.31.0, which only generates
kubeadm v1beta3 config where extraArgs is a string map. The
admission-plugin patch added in #586 uses the v1beta4 list form, so
kubeadm failed to parse the merged config and cluster creation broke
both integration workflows. kind v0.32.0 is the first release with the
v1beta4 template.

* fix(ci): force snapshot updates when building sample apps

The sample apps pin throwaway -SNAPSHOT builds of the dbaas clients, and
the Maven cache restored by setup-java carries snapshot metadata. With
the default daily update policy, a same-day publish-then-test cycle
reuses the cached snapshot instead of the one just published. -U makes
Maven re-resolve snapshots on every build.

* build(test-apps): consume spring client from main, quarkus from the rebuilt throwaway

Spring: the mounted-secret support merged to main (dbaas-client
9.1.7-SNAPSHOT, includes the DatabasePool cache fixes), so drop the
9.1.5-mounted-secret throwaway pin. Quarkus: the throwaway republished
at 10.1.5-mounted-secret-SNAPSHOT after merging the deduplicated feature
branch; its client classes now come from dbaas-client-base.

* build(test-apps): consume the Go dbaas client from main (#600)

The Go sample app was pinned to a throwaway feature-branch build of
qubership-core-lib-go-dbaas-base-client. The mounted-secret support has
merged to main (Netcracker/qubership-core-lib-go-dbaas-base-client#70),
so switch to the main pseudo-version (no release tag covers the merge
yet) and pick up the transitive qubership-core-lib-go 3.13.1 bump.

* fix(ci): keep per-pass Surefire reports in sample-service tests (#601)

All three sample-service passes (mount proof, REST fallback, direct M2M) run
`mvn test` in the same module, so each pass overwrote the previous pass's
target/surefire-reports. The final gate and the uploaded artifact only ever saw
the last pass, and the nightly job stayed green while pass A had been failing
12/12 on main since July 10.

Give every pass its own reports directory. Surefire 3.x exposes no user
property for reportsDirectory, so route it through a Maven property in the test
module's POM, the same way testFailureIgnore is already wired. The gate now
checks each pass's report set separately and treats a missing set as a failure,
so a pass that never wrote reports cannot pass silently. The diagnostics
trigger reads the per-pass directories too; the artifact glob already covers
subdirectories.

* build(test-apps): consume the quarkus dbaas client from main

The mounted-secret support merged to main
(Netcracker/qubership-core-java-libs#117), so drop the throwaway
10.1.5-mounted-secret pin — all three sample apps now consume the main
snapshots of their clients.

* feat(operator): report NamespaceBinding state in status (#599)

* feat(operator): report NamespaceBinding state in status

NamespaceBinding was the only CR without a status: registration was
visible only as a finalizer, a blocked deletion only as a Warning event
that expires after an hour, and a checker failure only in operator logs.
Give it the shared OperatorStatus and write it from the owning instance:

- Ready=True/BindingRegistered (phase Succeeded) once the finalizer is
  in place; Ready=False/BindingBlocked (phase Processing) while deletion
  is deferred; BackingOff/OwnershipCheckError when the blocking-resource
  check fails. Foreign instances keep writing nothing, so an unclaimed
  binding (operatorNamespace typo) is recognizable by its empty status.
- BlockingResourceChecker now returns the blocking kind names instead of
  a bare bool; the Ready message and the BindingBlocked event say what
  exactly is in the way.
- patchStatusOnExit tolerates NotFound: releasing the last finalizer can
  let the deletion complete before the deferred status patch runs.
- LastRequestID is assigned inside the defer — the finalizer patch
  refreshes the object from the API response and would wipe it.
- NamespaceBinding joins dbaas_resource_phase/condition/generation_lag
  for owned bindings; dbaas_namespace_binding_state stays as is.

Verified in kind: registered → Succeeded/True; EDB + delete → Processing
with 'deletion deferred: ExternalDatabase resources still present in the
namespace' and observedGeneration held back; EDB removed → binding
released.

* docs(operator): stop claiming NamespaceBinding has no status

The usage-examples section still said the finalizer was the single
indicator and argued a status field would add no value — contradicting
the status reference added in this PR and hiding the main diagnostic
for BindingBlocked and OwnershipCheckError. Point it at
status.phase/Ready, keep the finalizer as a supplementary signal, and
show how to read the blocking-kinds message when a deletion hangs. Add
the missing Status Reference ToC entry.

* fix(operator): stop amplifying workload churn into binding writes

Every workload watch event re-enqueued the binding, and after the status
change each such reconcile issued a status patch — LastRequestID alone
made the patch non-empty every time. During the concurrent IT section
that turned the binding into an API-write amplifier.

Filter the six workload watches down to create/delete events (spec and
status updates cannot change whether blocking resources exist), and skip
the exit patch when the status did not change, stamping LastRequestID
only on a real write.

* fix(operator): address binding status review findings

- A foreign finalizer can keep the binding alive after the protection
  finalizer is removed, leaving a BindingBlocked condition that names
  resources that no longer exist. The release path now rewrites the
  conditions to Ready=False/BindingReleased; when nothing else holds the
  object the deferred patch hits NotFound and is skipped as before.
- The six workload watches on NamespaceBinding fired on every update,
  so each binding status write re-enqueued all workloads in the
  namespace. Ownership only changes with create, delete, or a spec
  change — filter the watches with GenerationChangedPredicate.
- lastRequestId documented as the ID of the most recent reconcile that
  wrote the status: a steady-state no-op reconcile keeps the previous
  value, which the field's contract now says instead of contradicting.

* fix(operator): survive a failed release status patch

Removing the protection finalizer and writing BindingReleased happen in
one reconcile; when that status patch failed transiently, the retry hit
the finalizer-already-removed branch, changed nothing, and the stale
BindingBlocked survived forever on a binding held by another finalizer.
The branch now re-asserts the released conditions through the shared
markBindingReleased helper — a no-op write when the status already says
released. Regression test simulates the failing first patch.

Also document the BindingReleased state in the status reference and the
CRD godoc, which previously pointed only at BindingBlocked and
OwnershipCheckError for a hung deletion.

* docs(operator): list all NamespaceBinding reasons in the conditions godoc

The shared conditions godoc named only BindingRegistered and
BindingBlocked; BindingReleased and OwnershipCheckError were documented
in the howto table but missing from the API godoc and the generated CRD
descriptions.

* fix(aggregator): pin process-engine 1.5.5-SNAPSHOT with optimistic-lock race fixes (#603)

Picks up Netcracker/qubership-core-java-libs#151, merged to main and
published as 1.5.5-20260724.091358-10: atomic version guards on the task,
process, and context rows, conflict-recovering terminal saves, a single
completion callback per task, and a watchdog that no longer fails a task
that completed in time.

AbstractDbaaSTask.updateState switches from put+save to context.apply so
its status-description writes get the same conflict recovery; a plain
save() aborted the whole task when the POProcess tick or the timeout
watchdog wrote the same row first.

Verified on verify/process-engine-race-fix and verify/pe-race-fix-nb-status
against a throwaway snapshot of the same code: the DBaaS IT concurrent
section passes and the VersionMismatchException signature is gone from the
aggregator logs.

* chore: updating operator comments (#575)

* chore: updating operator comments

* chore(operator): regenerate CRDs after comment updates

The comment updates touched the shared Conditions field doc, which feeds the
generated CRD descriptions. Run make manifests and make sync-helm-crds so the
verify-generated job matches what is committed.

* docs(operator): clarify writeSecret convergence sequence

* docs(operator): correct status and polling comments

* docs(operator): address review feedback

---------

Co-authored-by: Roman Kichasov <roman.kichasov.qubership@gmail.com>

---------

Co-authored-by: Abhimanyu Roy <51357041+TheHollowMonarch@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants